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1.1 Research and Development (R&D) may be thought of either as a flow of 
spending or as a stock of accumulated knowledge from R&D expenditures 
which depreciates as new products or processes are introduced. The R&D 
conducted in one sector can have productivity-enhancing effects in the sector 
which performs the R&D either through cost reductions (process innovations) 


or market expansion (product innovations). 


1.2 Besides generating returns to the performing sector, R&D benefits can 


spill over to other sectors for three main reasons. 


e First, external benefits may occur because downstream users do not pay 
the full value of the input, as when banks purchase computers that are 

worth more to them than the price they pay for the machines. Also, quality 
improvements arising from R&D may not be reflected fully in higher prices 


because of competition in product markets. 


e Second, a technical discovery or an innovation can inspire work in 
another company. Competitors may copy or adapt the new technology. 
New ideas may emerge, new avenues of research may be undertaken, or 
previous results may become economical to pursue and bring to full 
fruition. For example, the invention of synthetic fibres by the chemical 


industry triggered many novel applications in the textile industry. 


e Third, research and engineering staff may leave to join other companies 


or set up their own, taking their knowledge with them. 


For all these reasons the innovating company cannot ‘appropriate’ all of the 
return to its own R&D: part of the benefit is obtained by its competitors, 
customers and employees and will not be captured. A distinction is thus made 
in the literature between private and social rates of return: economic returns 
which are appropriated by the R&D performer and returns which cannot be 


appropriated by the R&D performer but by society at large. 


1.3. R&D can also contribute to productivity growth indirectly through its 
interaction with other inputs. If capital and R&D are complementary, 
increasing the R&D stock may open up new possibilities for profitable capital 


investment. 
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1.4 R&D performed in-house and R&D performed elsewhere can also be 
complementary. Many authorities have stressed the dual role of R&D as the 
capacity to learn and to absorb. Although knowledge is a public good it cannot 
be absorbed costlessly. Instead, depending on the type of knowledge and the 
characteristics of the firm, a certain level of internal R&D has to be conducted 


in order to create an absorptive capacity. 


1.5 Finally, R&D can have indirect effects on productivity growth through the 
interaction of supply and demand. For example, R&D may reduce costs, which 
in turn lowers prices and increases quantities demanded. As demand and 
output expands productivity will be increased if there are increasing returns to 


scale. 
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20 


The presence of spillover effects thus poses difficulties for researchers 


wishing to estimate a rate of return to R&D spending. Even without spillover 


effects the causal connection between R&D and ultimate market effects is often 


imperfectly observed. Several difficulties have to be overcome if useful 


estimates of the rate of return to publicly funded R&D are to be obtained: 


one. 


e R&D is one input in a complex and uncertain innovation 
process: Other inputs are complementary and need to be included in 
descriptions of the production process; in general, there is no single model 


with which to assess and predict research results. 


e Public Sector outputs: Some public sector outputs cannot be priced, 
eg national security, improved policy advice. While it may be possible to 
price some public goods, the revenues cannot be appropriated from users; 
moreover, the cost of servicing additional users is near zero (eg R&D to 


help produce cleaner air) thus the merits of a pricing mechanism are 


doubtful. 


e Lags: R&D achieves returns at varying times in the future (see Table 1); 
high rates of discount will have the effect of reducing the present value of 
benefits to zero in many cases; in some cases discounting will have the effect 


of producing a negative net present value. 


In the public sector most Departmental outputs cannot be measured in 


monetary terms. Departments have become adept at using performance 


indicators to proxy outputs but, while these are useful, they cannot be equated 


with monetary outputs. 
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3. MEASUREMEN Tsy@r 
R&D VE TURIN Senin 
THE PRIVAREesSEG@R@R 


3.1. + Most of the empirical work in this field has originated in the USA. 
There have been two types of research procedure: case studies and econometric 


work. 
(I) CASE STUDIES 


oa These studies employ a variety of approaches; most of them measure 


returns by means of an ‘internal rate of return” unless otherwise stated: 


e Griliches (1958) estimated the economic rate of return to agricultural 
R&D in the USA over the 1910-55 period at about 35-40 per cent; 


e@ Mansfield’s (1977) work on 17 innovations in the US in the early 1970s 
estimated the private rate of return at about 25 per cent and the social rate of 


return at 56 per cent; 


e Mansfield’s (1991) work on the academic research results employed by 
76 US firms in seven industries indicated the rate of return at between 
22-28 per cent (per annum). This result has been queried by some 
observers (eg Smith, 1991) because of the rather crude methodology 
employed to obtain estimates (eg the cost denominator was the increment 


to R&D in one year rather than a depreciated stock of knowledge); 


e Robb (1991) of General Electric (GE) in the United States has reported 
one of many single firm studies: GE commissioned Booz Allen to survey 
190 “business transitions” dependent upon R&D over 1982-87 and 
estimated an internal rate of return of 20 per cent (with 50 per cent of the 


return obtained by only five cases). 


3.3. Studies of the returns to private sector R&D have sometimes been 
criticised for producing inflated results because the cost of complementary 
inputs have not been included in the calculation of returns. On the other hand, 
most such studies cannot capture all the external (or spillover) benefits of 
research, because they are in unrelated fields, or because the innovator is too 
far from the ultimate beneficiary in the production->consumption chain. This 


may result in an under estimate of the returns. 


* — The discount rate which, when applied to the future stream of net benefits, provides an exact balance between initial R&D costs and 


prospective net benefits, 
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3.4 It might be considered instructive to contrast these private sector rates of 
return to R&D with rates of return obtainable from other investments, for 
example, in respect of fixed capital, marketing facilities, etc. Unfortunately, 
there is no reliable information on a range of alternative investments. There are 
data on typical hurdle rates in British industry. For example, Pike (1982) found 
that 40 per cent of respondents used a hurdle rate of 20 per cent or over for 
normal risk projects. Scapens and Sale (1981) found average rates of 18.5 per 
cent in the UK and 17.1 per cent in the US in their survey. Also, in the opinion 
of Hayes and Garvin (1982), hurdle rates ‘are typically quite high, often in the 
range of 25% to 40%, and there is some evidence that they have been rising over 
the past decade’. But such hurdle rates are essentially ex ante barriers intended 
to screen out poorly performing investments rather than indicators of ex post 
rates of return. They usually include allowances for risk, inflation, and some 
non-yielding overhead investments that have to be ‘carried’ by positive yield 


investments (Coulthurst, 1986). 
(II) ECONOMETRIC WORK 


3.5. In general, researchers estimate a multi-variable production function in 
order to take into account the impact of several inputs, including R&D. 
Griliches (1986, 1987) the main exponent of this approach finds that: (i) the 
stock of R&D capital contributes significantly to cross-sectional differences in 
productivity between sectors; (ii) the effect of basic research in the private 
sector is positive and significant; (ii1) privately financed research has slightly 
higher returns than federally funded research. On average the rate of return 
equivalents estimated from Griliches’ work were 51 per cent for 1967 and 62 
per cent for 1972, reasonably comparable with the social returns obtained from 


Mansfield’s early work. 


3.6 Griliches summarises the vast range of literature in this field as 
indicating a gross rate of return in the range 20-50 per cent. Méhnen (1990) 
confirmed two of Griliches’ results. First, there is a higher rate of return for 
basic R&D compared with applied or ‘development’ R&D. This finding is not 
surprising: private firms perform relatively little basic research, and there are 
more opportunities for spillover benefits from basic research. Moreover, basic 
research costs are relatively low compared with applied research and 
development costs; and, once it is clear that a line of enquiry is not promising, 
projects can be terminated without demonstrable adverse effects (Morris et al, 
1991). Second, there is a higher rate of return on company-financed as opposed 
to publicly financed R&D. Again, this result is not unexpected because much 


publicly financed R&D is in areas where the risks involved are relatively high. 
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3.7 Several studies suggest that inter-industry differences in returns to R&D 
are significant; and that such differences tend to be greater than inter-country 
differences. Ranking industries in descending order of rate of return from 
R&D (based on the US and Canadian literature) shows the following: rubber and 
plastics, scientific instruments, petroleum, non-electrical machinery, primary 
metals, chemicals, gas and oil exploration, electricals, metals fabrication, food 


and beverages, pulp and paper, transport equipment, and aircraft and parts. 
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4. SUMMARY OF 
PEPE RATE PEN DINGS 
COSINE EAR aM dey ede OTR RU Resa B. 


4.1. R&D has been identified as an important determinant of productivity 
growth. The evidence suggests that the private (net of depreciation) rate of 


return lies in the 10 per cent to 40 per cent range. 


4.2. R&D is toa large extent a public good that may be consumed by other 
firms in the same industry, and by other industries in the same country; it can 
also spill over country borders. These spillover effects can be substantial. Social 
rates of return 50 per cent to 100 per cent in excess of the private rates of return 


are not uncommon. 


4.3. The mix of R&D is of some importance. It is now accepted that higher 
rates of return are obtained from company-financed R&D rather than 
publicly-funded R&D; this may be because the objectives of publicly-funded 
R&D are systematically different from the aims of privately financed R&D. 
Social rates of return to industrial R&D are highest in sectors such as chemicals, 
non-electrical machinery or scientific instruments. Policy-makers should not be 


insensitive to the type of R&D being funded or encouraged. 
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5.1. The economics literature has approached the assessment of returns from 
R&D at the aggregate level by asking the question: what is the relationship 
between technological change (directly and indirectly measured by R&D) and 
economic growth? The earliest studies, by US economists studying growth in 
the first half of the present century, suggested that growth arising from 
technological change was about 1.5 per cent per annum. On the basis of this 
finding, it was concluded that about 90 per cent of the increase in output per 
capita during this period was attributable to technological change. The basic 
methodology used in these early studies was as follows: the total output of the 
economy arises from various inputs of productive services into the productive 
process; these inputs can be identified as labour and capital and attempts may be 
made to estimate their contribution to the measured growth of output; 
whatever portion of the measured growth of output cannot be explained by 
these inputs may be attributed to technological change. The crudeness of this 
procedure is transparent. The resulting measure of the effect of technological 
change contains the effects of whatever inputs are excluded which may be 
increases in economies of scale, an improved allocation of resources, increases 


in education, or improved health and nutrition of the labour force. 


5.2. Later work has gone some way to remedy this shortcoming. For 
example, Denison (1967) included many inputs - particularly changes in labour 
quality associated with increases in schooling - omitted in earlier studies. 
Denison concluded that technological change, or what he termed the “advance 
of knowledge”, accounted for about forty per cent of the total increase in US 
national income per person employed during 1929-1957. Subsequently, he 
estimated that the advance of knowledge was responsible for 1.4 percentage 
points of the annual growth rate of national income per person employed 
during 1948-1969, and for 1.6 percentage points of its growth rate during 
1969-1973. It is acknowledged that this body of work still has some defects. 
Nevertheless, based on the available evidence, technological change seems to 
have been a very important factor, perhaps the most important factor, 


underlying long-term economic growth in the United States and elsewhere. 


5.3. This work has generally been taken to undergird the case for public 
sector funding of R&D. Because aggregate studies have shown a healthy 
positive effect of technological change on economic growth, and because the 
public sector has supported investments in technological change (as well as 
investing heavily in the “advance of knowledge”), Governments have argued 
with some force that their R&D investments have contributed to wealth 


creation. 


5.4 In contrast, public sector microeconomic studies are pessimistic about 


the possibility of estimating returns to R&D at the departmental or programme 
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level. For example, Paul (1992) distrusts the calculation of economic returns 
even within industry where research is sharply focused on processes or 
products. Given the time lags and the absence of reliable data on supplementary 
inputs (that is, the types of inputs used by Griliches and others in econometric 
work) Paul states: “... typical cost benefit analyses can, at best, be indicative and, 
at worst, spurious.” He also takes the view that “... attempts to evaluate the 
work purely on ex-post financial criteria may not only be inaccurate but also 
misleading.” However, Paul recommends the use of non-financial allocation 
criteria which are reasonably systematic: scoring models, return on investment 
in specific industry cases where reliable market data are available, and research 
effectiveness indicators (see Section 6.). These may be used as indirect 


indicators of the rate of return. 


5.5. In the USA an Office of Technology Assessment (OTA, 1986) study 
came to broadly the same conclusions. The OTA suggests three fundamental 


reasons why precise measurement of R&D outputs is difficult: 


(i) Non-economic aims, such as defence, where the aims are to encourage 
socially desirable, high risk investments which the private sector will not, or 


cannot, handle; 


(ii) Problems of benefit measurement, for example, in health advances, 
where realistic market prices are not set; consequently returns do not accrue 


to the investor and tend to spill over into other sectors; 


(iii) Many outputs are available at zero marginal cost and zero price; to 


collect revenues would cost more than the cost of production. 


5.6 | The OTA believes that Governments cannot employ a private sector 
methodology to assess monetary returns to publicly funded R&D. However, 


the OTA admits exceptions, namely: 


(i) agricultural R&D: see above Griliches (1958). These findings were 
replicated by many others (see Table 2). However, some agricultural 
economists are less sanguine about the work in this area since the 
environmental impacts of agricultural advance have become more 


pronounced. 


(ii) aviation: where Mowery (1985) has estimated internal rates of return 


in the range 24-27 per cent; 


(iii) health: where Mushkin (1979), using a human capital approach, 


obtained an internal rate of return in the 40-50 per cent range. 
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5.7. One other point made by the OTA is worth noting here. It is that all of 
the production function models employed to estimate returns to R&D, both in 
the public and private sectors, do not necessarily demonstrate causality. For 
example, the finding obtained by Griliches in respect of basic research in the 
private sector might be explained by rich successful companies being able to 
indulge themselves in basic research. However, it is not possible to dismiss 
such a large body of work out of hand by alleging reverse causality. So many 
researchers have replicated these results (including some who have tested for 
reverse causality), and so many private sector firms invest in R&D with the 
clear expectation of gain, that the argument concerning reverse causality is very 
weak . As the Organisation for Economic Co-operation and Development 
(OECD, 1991) has observed: “It is therefore an agreeable surprise to discover 
that most studies have managed to produce statistically significant and 
frequently plausible estimates of the elasticity of R&D or the rate of return to 
R&D. Individual case studies and other factual knowledge in the field, as well 
as the fact that firms do indeed undertake research, leave little room for doubt 


on this score.” 


5.8 For the UK, there is scattered evidence of the use of cost benefit analysis 
(CBA) techniques in Departments, but the same problems noted by the OTA 
and Louis Paul have been encountered. There is a wealth of evidence that 
departments have employed CBA to assess (ex ante) or evaluate (ex post) 
investment programmes: in agriculture, forestry, flood defence systems, 
fisheries vessels, the Third London Airport, road improvements, office 
relocation, overseas aid projects, etc; and cost-effectiveness studies are a 


commonplace in health and defence. 


5.9 However, these are not all areas where R&D is very important. In 
Health, R&D can play a major part in preventing or reducing disease but there 
are still undecided questions about the valuation of benefits; and clinical 
research has uncertain pay-offs over long periods. There have been attempts at 
relating agricultural R&D to productivity improvements, both in the Ministry 
of Agriculture, Fisheries and Food (MAFF) and in the National Institute of 
Agricultural Engineering. These attempts have never proceeded to specify an 


overall departmental rate of return. 


5.10 One interesting approach in the public sector is that of the Bureau 
d’Economie Théorique et Appliqué (BETA) group in Strasbourg. For example, 
this group has sought to calculate the indirect economic effects of the European 
Space Programme (Bach and Lambert, 1993). The methodology distinguishes 
between technological, commercial, organisational and work factor effects and 


quantifies these in terms of ‘added value’ (meaning the sum of the wages and 


RETURNS TO RESEARCH AND DEVELOPMENT SPENDING 


profits of firms receiving contracts from the European Space Agency (ESA)). 
Managers were asked to estimate co-efficients which attributed the contribution 
made to sales by the four effects listed above and estimated the contribution of 
ESA work to those effects. The process was extended to firms which supplied 
the contractors. A figure was produced which represented the ratio between: 
the total value of indirect effects generated by the ESA contractors, and the total 
payments made by ESA to those contractors. Criticisms of the method have 
focused on its potential for subjectivity, the failure to consider investments 
needed after the technology is developed, and the failure to discount future 
benefits. The approach has recently been applied to two of the EC’s 
programmes (Basic Research in Industrial Technologies for Europe and 
European Research in Advanced Materials (BRITE, EURAM)) and yielded a 
highly positive ratio. 


5.11 Finally, it is worth noting that defence R&D spending has been assessed 
in terms of the defence industry’s contribution to output, employment and the 
trade balance. For example, Table 3 shows how the UK aerospace industry, 
and the trade in military aircraft and parts, contributed to the visible trade 
balance during the period 1981-91. It is clear that over the decade the UK 
aerospace industry was in surplus with the rest of the world and there are 
grounds for suggesting that the trend in the balance was slightly positive. 
However, these data do not imply that the rate of return on aerospace R&D 
investments was positive. It seems likely that the rate of return on privately 
funded aerospace R&D was positive; but support for aerospace R&D in the 
form of Launch Aid (payments net of the present value of Launch Aid receipts) 
will have increased the total spend on R&D. To that extent, the rate of return 
on total aerospace R&D spending, from both private and public sources, will 


have been reduced. 


5.12 Insummary, broad assessments of the return to R&D spending by the 
public sector are possible at the aggregate level. And at the individual project 
level there are several examples, both in the UK and overseas, of reasonably 
firm estimates of returns. However, at the intermediate level of the overall 
programme, or Departmental, spend the problem of attributing returns to an 
eclectic collection of projects, many of which have long gestation periods or 


non-financial policy objectives, has posed grave difficulties for research. 
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INTRODUCTION 


6.1 The previous sections reviewed attempts to assess quantitative ex post 
rates of return to R&D; the subject of ex ante assessment is now discussed. In 
one sense, ex ante assessment must be performed before almost any decision is 
taken on the allocation of resources to R&D. This can take place at several 
different levels of aggregation ranging from national technology priorities, to 
the organisational, programme or portfolio level, and to the level of project 


selection. Approaches too may vary, addressing issues such as: 
e how much R&D to perform; and 


e how to allocate resources by some mix of informal judgement and 


decision aids. 


Here the focus is primarily on more systematic approaches, in particular those 
which are concerned with investment decisions. Inevitably, much of the 
experience in this area has been accumulated in the private sector where almost 
all R&D spending is made with a view to economic return. However, ex ante 
assessment is also being given increased prominence in the public sector. Some 
of the methods and techniques available to firms are described and their 
strengths and limitations are discussed, together with some new developments. 


The report concludes with some lessons for government R&D. 


PUBLIC SECTOR EXPERIENCES 


APPRAISAL 


6.2. The practice of performing a formal appraisal of an R&D programme or 
project before it is selected is now well established in UK government. As 
defined in the Cabinet Office publication, R&D Assessment - A Guide for 
Customers and Managers of Research and Development, programme appraisal is 
directed at establishing whether there is a need for the R&D concerned, 
defining the programme and its objectives, and culminating in a statement of 
justification. The most widely used approach is one initially developed in the 
Department of Trade and Industry, the ROAME system. ROAME (an 
acronym for Rationale, Objectives, Appraisal, Monitoring and Evaluation) is a 
discipline which seeks to establish rationale and objectives and to state how 
programmes will be appraised, monitored and evaluated. It is relevant to this 
discussion because it could provide a suitable framework in which to locate 


appraisal of the likely economic returns to R&D spending. 
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6.3. The ROAME style approach can be complemented by the use of logical 
frameworks which make more explicit the connections between objectives and 
expected outcomes and often specify verifiable levels of attainment. Logical 
frameworks were originally applied for development projects and their use is 


most extensive in the Overseas Development Administration. 


6.4 Approaches of this type are also used overseas, notably in Canada where 
evaluations of government-funded R&D make extensive use of the ‘logic chart’ 
approach which relates programme objectives and activities with upward links 
to higher level objectives and downward links to expected impacts. Recently, 
the emphasis in Canada has been on the development of ‘performance 
expectations’, an approach which defines strategic and desired levels of 


performance with a direct link to the objectives of the programme. 


CRITICAL TECHNOLOGIES EXERCISES 


6.5 Ata higher level of aggregation recent efforts in several countries to 
identify ‘critical technologies’ have entailed an element of economic assessment, 
typically involving an intersection between technological promise and potential 
market impact. For example, the approach adopted by the Australian 
Commonwealth Scientific and Industrial Research Organisation (CSIRO), 
subsequently replicated in New Zealand, aimed to assess the potential benefits 
to Australia of research in different areas and to identify factors relevant to the 
achievement of those benefits. CSIRO used four criteria in assessing whether a 


technology was critical: 


e the potential economic and social benefits; 

e Australian capability to achieve those benefits; 
e R&D potential and costs; 

e Australian research capability. 


The Australian study incorporated ways of scoring to enable two 
market-oriented parameters to be derived, attractiveness (benefits and the 
potential to achieve them) and feasibility (R&D potential and capability). 
Although outside information was used, the expertise drawn upon was 
primarily from within CSIRO. The outcome was presented in the form of a 
graph which enabled the position of the selected technologies to be seen readily 
and a form of ranking to be set out. This study and other exercises represent a 
means of ordering expert opinion in a systematic manner; they do not attempt 


to provide a precise measurement of the rate of return. 
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6.6 One feature of the CSIRO approach which might be stressed in any 
United Kingdom procedure is the assessment of regional appropriability of 
benefits. It may not be sufficient simply to identify potential returns from a 
piece of R&D; it may also be necessary to demonstrate that these returns would 
accrue to the United Kingdom. This would, for example, exclude benefits 
arising from research in an activity where industrial capability lay exclusively 
overseas and where barriers to entry made it unlikely that UK industry would 


be able to enter the market. 


6.7. Another lesson to be derived from the experience of critical technologies 
exercises is that most of the technologies selected are inter-related and 
pervasive, that is they underpin a wide range of other technologies and 
economic activities. Indeed, this pervasive characteristic is one reason why 
public support is justified. It follows that any attempt to estimate the benefits 
arising must take account of these relationships. Matrix methodologies have 
been developed to track these relationships and identify pervasive areas of 


science and technology (Lowe and Georghiou, 1989). 


COST BENEFIT APPROACHES 


6.8 Cost benefit approaches to R&D resource allocation in the UK public 
sector were last carried out to any significant extent in the 1960s and early 
1970s. The focus of activity was the Programmes Analysis Unit (PAU), located 
in the Ministry of Technology. The PAU approach aimed to assess R&D 
projects for programmes against the criterion of national benefit, including 
financial returns but extending more broadly. The basic criterion was the 
expected benefit/R&D cost ratio, qualified by the perceived risks and broader 
non-quantifiable or intangible factors. Quoting a retrospective review of the 
PAU’s work, P M S Jones a former member of the Unit, lists a number of 
benefits arising from appraisal, including clarification of objectives, 
identification of options and ‘guidance on the balance between parts of 
programmes and on the effect of government support or non-support’ (Jones, 
1989). However, the review concluded that quantitative appraisal could not 
give guidance on the proper total level of governmental or departmental R&D 
expenditure nor offer much general guidance on the split of budget between 
widely different sectors of activity. Jones echoes some of the statements made 


about ex post calculations of returns to R&D: 


“Efforts to combine disparate costs and benefits into a single objective function conceal 
value judgements: multi-attribute analysis in which trade-offs are fully exposed, 


including those that defy quantification, are to be preferred.” 
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Jones reviews more recent work in cost-benefit analysis (for example the 
Sizewell inquiry) and concludes that cost-benefit analysis can provide valuable 
insights and perhaps clear guidance on small scale activities but is too prone to 
uncertainty where large scale projects or programmes with potentially 


wide-ranging impacts are concerned. 


6.9 Similar conclusions were drawn by Norris and Vaizey (1973) and Byatt 
and Cohen, (1969). The approach of Norris and Vaizey was basically similar to 
the more recent work by Mansfield (reported above in Section 3). It started 
with the notion that basic research may lead to new industries, or may have 
smaller scale applications in industry. The method was to see what effect a 
notional delay in the scientific discovery would have had on the net benefits of 
the discovery. Net benefits were defined as the change in output less the costs 
of applied research, development, investment and manufacturing. A feasibility 
study of this approach found that although in some cases the economic benefits 
of a scientific discovery could be traced, in general it was not feasible because 
the interaction between science and technology could not be adequately 


described by the linear model assumed in the method. 
PRIVATE SECTOR APPROACHES 


6.10 In private companies R&D usually competes for resources with other 
forms of investment, although in certain sectors, such as biotechnology, new 
companies may be established with venture capital funds. Within the overall 
budget for R&D individual project choices for both planned and ongoing 
research need to be prioritised, again broadly in terms of investment criteria. 
These choices are, of course, restricted by management perceptions of the 
strategy and ‘core competences’ of the firm (Prahalad and Hamel, 1990). In the 
following paragraphs the application of quantitative techniques at project and 
portfolio level is examined; then the broader environment in which firms seek 


to gain economic benefit from their R&D activities is considered. 
PROJECT SELECTION 


6.11. There is a large body of management literature on the application of 
quantitative techniques to the selection of R&D projects. Much of this 
literature comes from the 1960s and 1970s. Baker (1975) classified the 
approaches used in this literature. What he terms ‘objective functions’ are 
defined as: ‘systematic procedures for soliciting and integrating subjective and 


objective benefit data’. 
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These were placed in three categories: 


e Comparative models - which require the respondent(s) to compare 
one proposal either to another proposal or to some subset of alternative 


proposals; 


e Scoring models - which require the respondent(s) to specify the 
merit of each proposal with respect to each of several project characteristics 
(criteria), whereupon criterion scores can be aggregated to yield an overall 


project score; and 


e Benefit contribution models - which require the respondent(s) to 
tie the projects directly to R&D objectives or to system requirements, and 
benefit is measured in terms of contribution to the objectives or to the 


requirements. 


The experience of companies in applying project selection techniques has been 
reviewed on more than one occasion (see Baker, 1975; Liberatore and Titus, 
1983; Brockhoff and Pearson, 1992). All of these reviews draw a similar 
conclusion: that industry has not extensively employed these techniques. An 


exception to this is the use of scoring models and related techniques. 


6.12 The OTA study (1986) in its review of documented experience 


identified four categories of formal techniques. 
I) SCORING MODELS 


Corresponding to Baker’s category above, this involves rating each project 
against a series of relevant decision criteria. Scores for each project are 
combined to give a single project score. Criteria may be economic or 
non-economic (eg technical merit). Approaches of this type are widely used in 
UK Research Council committees. Evaluations have noted that criteria relating 
to scientific quality normally outweigh potential economic return in the final 
ratings, although there is rarely a formal weighting system. It appears that 
scientists and technologists (including industrial members) dislike supporting 


projects which may not be of the highest technical quality on strategic grounds. 
6.13 Paul, reporting on his experience with Shell R&D (Paul, 1992), offers a 


more positive view of scoring models as an approach to ex ante evaluation. His 


basic condition is that the peer panel which assigns the scores should represent 
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not only the peer panel of researchers but also those close to the market and 


product. Furthermore he insists: 


e Projects have to be based on a need identified by a customer. Following 


in-depth discussion objectives need to be defined to facilitate checking of 


progress; 


e Projects should be classified as Basic, Applied or Development and those 


categorised as Basic should not be assessed on financial criteria; 


e Applied and Development projects should be prioritised by peer panels 


constituted as above. 


He suggests the use of a matrix of criteria (derived from the Boston Consulting 
Group Approach) which compares attractiveness and feasibility on similar lines to 


the CSIRO approach described above (see Figure 1). 


6.14 Other techniques have been used to add rigour to scoring approaches. 
Q-sorting was suggested in 1975 (Helin and Souder, 1975), whereby a group of 
experts directly prioritise various projects. More recently, an approach known 
as QFD (quality function deployment) has moved on to the use of a peer group 
to prioritise the criteria for evaluation. The technique uses matrices and decision 
trees to facilitate communication and simultaneously define and document 


customer requirements (Bossert, 1991). 
Il) ECONOMIC MODELS 


These approaches rate projects against a series of economic criteria such as 
expected rate of return. The basic approach is one of capital budgeting or 
investment appraisal. Each indicator (eg Net Present Value, Internal Rate of 
Return, Return on Investment, etc) produces a quantitative measure of the 
attractiveness of the investment, normally discounted over time. An allowance 
for risk may be added independently. Despite the apparently precise economic 
formulation it should be stressed that the data for an approach of this type are 
often derived from expert opinion and are no more intrinsically valid than 
scoring models (which in effect they are). In addition, the different indicators 
(which should be used in parallel on any single appraisal) may give conflicting 
advice, for example in the rank ordering of a set of independent projects. 
Critical assumptions need to be made in adjusting for time and risk. Graves 
and Ringuest (1991) showed that the technical combination of these 


adjustments can produce biases, for example against shorter or longer term 
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projects. Norris and Vaizey were even more critical of such approaches. Their 


objections may be summarised as follows: 


e Asreturns to R&D are typically longer term compared with other forms 


of investment, they will be heavily discounted and count for very little; 


e Risk and uncertainty, particularly in fast-moving areas, will be 


substantial and very difficult to calculate. 


III) CONSTRAINED OPTIMISATION OR 


PORTFOLIO MODELS 


These models examine a mix of projects rather than individual cases and use 
programming techniques to allocate resources among the candidates. 
Optimisation is against economic objectives with specified resource constraints. 


High quality data are necessary. 


IV) RISK ANALYSIS OR DECISION 


ANALYSIS 


STRATEGIC 


These analyses require high quality data. Such models describe the expected 
utility of alternative budget allocations among a set of research projects, making 
use of probability distributions. As noted earlier, the use of purely quantitative 
approaches in industry is rare, although many larger firms will go through some 
less formal version involving scoring and/or investment appraisal (for example, 
British Gas uses discounted benefit/cost ratios to prioritise). Few would regard 


the outcomes of these as an automatic means of selection. 
APPROACHES 


6.15 The above discussion focused on project selection. This approach has 
been criticised. For example, Pearson (1975) has pointed out that these exclude 
alternative approaches to a problem or project opportunity which might be 
available but remain unidentified. Work on the accountancy treatment of R&D 
(Nixon, 1991) pointed out that evaluating R&D expenditures on a ‘project by 
project’ basis ignores the reduction in uncertainty associated with undertaking a 
portfolio of R&D projects. Nixon argued that the disaggregated approach leads 
to a consistent over-estimation of uncertainty which in turn leads to an 


inappropriate treatment of R&D in accounting procedures. 
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6.16 Norris and Vaizey nonetheless considered it worthwhile to carry out an 
economic evaluation because it forces people to think through the implications 
of a proposal. Furthermore, they emphasised that all branches of a company 
should take part in the decision. This early conclusion provides a bridge to 
current thinking in R&D management studies which see economic evaluation 
of anticipated benefits as part of a process which emphasises the importance of 
communications both between different functions within the company and 
with its outside networks of suppliers, customers, collaborators and 
competitors. Hence, the practice in a Japanese corporation of requiring its 
researchers to estimate the economic returns to their projects: not because the 
results will be used, but to ensure that the researchers have a business 
orientation and can communicate with production departments of the 


company. 


6.17 With this is mind it is possible to reappraise the value of some of the 
techniques considered. Matrices and the use of scoring models can be seen as a 
means of bringing together personnel from different backgrounds and 
providing them with a channel of communication, thus facilitating horizontal 
relationships across the organisation. Research planning diagrams can be used 
to develop a consensus about the requirements of an R&D project and to help 
identify and anticipate problems both within the project and in the 
implementation or commercialisation process. As mentioned above, 
approaches such as quality - function-deployment (QFD) can extend to 
involving the customer. Risk management procedures can be used to enhance 
these approaches, with the emphasis on identifying what is not known about a 


project and quantifying those risks. 


6.18 Other recent techniques focus on external networks. To be able to learn 
and benefit from R&D conducted elsewhere needs an R&D capability which is 
sufficiently ‘state of the art’. An R&D project can thus be seen as an ‘option’ to 
become involved in a technology at a later date. This approach leads to the 
potential application to R&D of option-pricing theory, as used in financial 
markets. This technique has been explored by some large companies but little 
information is available in the literature which is specific to R&D (Newton, 


1991). 


6.19 An indication of the factors industry considers significant in answering 
the question ‘How Much R&D?’ is given in a paper of the same title 
(EIRMA-European Industrial Research Management Association, 1992) which 
reported on a workshop and questionnaire findings. First, it notes that feedback 
on performance is normally too late to influence subsequent allocative 


decisions. Second, some influences on R&D levels are driven by political, 
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social and technological factors beyond the control of the business. For 
example, there is political pressure for improved environmental performance. 
Third, R&D is getting closer to the customer, both geographically and 
organisationally. Finally, while profit growth is a strong determinant of 
perceptions of R&D intensity for the future, firms recognise that future 


investment is also influenced by current profits and the ability to pay. 


6.20 Allocation of resources is seen by EIRMA as a matter of achieving 
balanced top-down (corporate) and bottom-up (business) allocation. Again, 
emphasis is placed on organising R&D to align with business and markets with 
effective project selection and feedback on the benefits achieved. 
Managing the total innovation process is as important as selection of projects: 
much work has been done, for example, on the role of product champions and 
on organisational structures. Considering which companies have the capability 


to appropriate the benefits is another important consideration. 


6.21. The lesson which might be drawn for publicly-funded R&D here is that 
whatever the utility of ex ante assessment techniques, they are a necessary but 
not a sufficient instrument to achieve greater economic benefits from R&D. 
Achievement of that objective would, in addition, require organisational change 
involving the managers of the R&D and their relationship with the firms which 


would ultimately be responsible for realising those benefits. 
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6.22 Given the uncertainty involved in forecasting the future, and the 
complexity of the innovation process, it is hardly surprising that ex ante 
appraisal of R&D is even less likely than ex post evaluation to give accurate and 
detailed information on rates of return such that choices may be made across 
widely different areas of R&D. Nonetheless there are a number of useful 
lessons to be derived from private sector practices in this area which offer scope 
to put R&D decision-making in the public sector onto a more rational basis 
and, more importantly, to increase the likelihood that there will be an economic 


return to Government-funded R&D. The following points can be made: 


e Current best practice for assessment in government R&D requires that 
programmes should undergo a systematic appraisal to ensure that there is a 
rationale for support and that objectives are clear. In support of this 
approach, one possibility would be, for example, taking into account 
economic benefit to the United Kingdom as a horizontal consideration, 
wherever possible, for Government R&D; and for this to feature in appraisal 
exercises. This is not to say that there should be any diminution of the 
other objectives, but rather that those performing an appraisal should 
include an explicit statement about expected economic consequences and 
the appropriability of R&D results. In some cases these may be zero or even 
negative but outweighed by other benefits. This requirement would have 
the benefit of focusing programmes and perhaps re-orienting them towards 


taking advantage of opportunities for wealth creation. 


e Inconsidering the return to R&D the question of appropriability 
should always be in mind. It may not be sufficient to identify an area as 
being of high promise - there should be a reasonable prospect of its being 
exploited in the United Kingdom to a sufficient level to justify the 


investment. 


e There are a number of techniques available for structuring expert 
opinion and for applying the investment appraisal metaphor to R&D. None 
offers a unique solution but most could help to organise thinking and 
improve decision-making. The use of scoring models in industry is 
testimony to this. Those performing appraisals should be encouraged to use 
these approaches, perhaps as a ‘battery of tests’ in order to inform the 


exercise. 


e Many of the problems encountered in these approaches stem from a lack 
of adequate data. Systematic analysis of past performance is one way to 
improve this situation. This reinforces the need to treat assessment as a 


cycle with evaluation informing appraisal. 
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e The investment metaphor can also usefully be applied to R&D which is 
directed at non-monetary benefits such as improved health, safety and 


welfare generally. 


e Fostering an investment-oriented approach among those performing 
appraisals is unlikely to be achieved solely by production of guidelines. An 


active strategy of training and support is needed. 


e In the final analysis, if the objective is not simply to measure but is also 
to optimise and increase the contribution which government-supported 
R&D makes to wealth creation, then it is insufficient to reallocate that R&D. 
In industry, project selection has come to be seen as a small subset of 
strategic and organisational issues. This is also true for Government. 
Economic benefit can only be realised through innovation and 
commercialisation. A strategy for increasing this requires close examination 
of the entire organisational structure of Government and public sector 
institutions, with particular emphasis on the linkages to industry and 


sources of finance. 
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TABLE 1 INDUCTION TIMES FOR EXPLOITATION 
INTERVAL BETWEEN INTERVAL BETWEEN 
INVENTION AND INVENTION AND 
EXPLOITATION EXPLOITATION 
PRODUCT (YEARS) PRODUCT (YEARS) 
Safety razor 2 Jet engine 
Fluorescent lamp 79 LP record 3 
Vi 22 Magnetic recording 5 
Wireless telegraph 8 Perspex 3 
Triode vacuum tube ii Nylon 11 
Spinning jenny 5 Power steering 6 
Watt steam engine 11 Radar 13 
Ball point pen 6 Streptomycin 
DDT 3 Terylene 
Freon refrigerant 1 Xerography 
Gyro compass 56 Zipper 
Hardening of fats 8 


Source: J. D. Birchall, FRS. 
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TABLE 2 ECONOMETRIC STUDIES OF THE PRODUCTIVITY RETURN 


AGRICULTURAL RESEARCH IN THE UNITED STATES 


AUTHOR (DATE) COMMODITY TIME PERIOD RATE OF RETURN 
Griliches (1964) Aggregate output 1949-59 
Peterson (1967) Poultry 1915-60 
Evenson (1968) Aggregate 1949-59 
Cline (1975) Aggregate 1939-48 
Knutson and Tweeten| Aggregate 1949-58 
(1979) 1959-68 
1969-72 
Bredahl and Peterson | Cash grain 1969 
(1976) Poultry 1969 
Dairy 1969 
Livestock 1969 
Davis (1979) Aggregate 1949-59 
1964-74 og 
Evenson (1979) Aggregate 1868-1926 65 
1927-50 95 (applied R&D) 
1927-50 110 (basic R&D) 
1948-71 45 (basic R&D) 
Davis and Peterson Aggregate 1949 100 
(1981) 1954 79 
1959 66 
1964 37 
1969 zo 
1974 a7 
Norton (1981) Poultry 1969 i) 
Dairy 1969 56 
Livestock 1969 30 
Cash grain 1974 44 
Poultry 1974 33 
Dairy 1974 66 


Source: RD Weaver, “Federal R&D and US Agriculture: An Assessment of the Role and Productivity Effects”, paper presented at the 
National Academy of Sciences Workshop on “The Federal Role in Research and Development”, Nov. 21-22, 1985, p.27. 
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TABLE 3 UK BALANCE OF TRADE IN AEROSPACE AND MILITARY AIRCRAFT 
VISIBLE TRADE BALANCE MILITARY AIRCRAFT AND PARTS 
(£M) (£M) 
UK UK 
YEAR ECONOMY AEROSPACE EXPORTS IMPORTS BALANCE 
INDUSTRY 
1981 6252 831.4 183 
1982 1910 1125.1 216 
1983 -1537 1016.5 260 
1984 -5336 857.1 212 
1985 -3345 1061.4 174 
1986 -9559 1931.6 192 
1987 -11582 237103 326 
1988 -21480 1690.6 813 
1989 -24683 3222.5 1524 
1990 -18809 1664.3 1206 
1991 -10290 2501.7 1367 


Sources: | SBAC; Monthly Digest of Statistics; Defence Statistics 1992, Ministry of Defence. 
Notes: (i) UK economy figures are seasonally adjusted 


(ii) Exports and imports for the aerospace industry comprise all military and civil aerospace products. The trade in military aircraft and parts 
does not include guided weapons, missiles and parts, nor does it include the group classified as additional aerospace equipment. 
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FIGURE 1: A TYPICAL SCORING MODEL 


ATTRACTIVENESS 


IMPERATIVE 


VERY ATTRACTIVE 


MODERATELY ATTRACTIVE 


BARELY ATTRACTIVE 
NOT ATTRACTIVE 
FEASIBILITY => 
CRITERIA: 
ATTRACTIVENESS FEASIBILITY 
- Importance of the technologicalperformance - Fertility of research field 
relative to the market - Scientific competence 
- Size of the market (or market potential) - Position of current technology on 
- Projected market growth “S-curve” 
- Amount of environmental damage avoided - Chances of success 


- Enhanced social amenity 


- Health and safety improvement 


Source: Paul, (1992) 
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